Active Token Mixer

The three existing dominant network families, i.e., CNNs, Transformers and MLPs, differ from each other mainly in the ways of fusing spatial contextual information, leaving designing more effective token-mixing mechanisms at core backbone architecture development. In this work, we propose an innovative token-mixer, dubbed Active Token Mixer (ATM), to actively incorporate information tokens global scope into given query token. This fundamental operator predicts where capture useful contexts learns how fuse captured with token channel level. way, range can be expanded a limited computational complexity, way is reformed. We take ATMs as primary operators assemble them cascade architecture, ATMNet. Extensive experiments demonstrate that ATMNet generally applicable comprehensively surpasses different families SOTA vision backbones by clear margin on broad tasks, including visual recognition dense prediction tasks. Code available https://github.com/microsoft/ActiveMLP.